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Technical  Objectives 

Grant  #  F49620-00-1-0211,  entitled  Facility  for  Local  Postprocessing,  Visualization  and  Animation  of 
Remotely  Simulated  Very  Large  Temporal  Datasets  was  a  DURIP  equipment  Grant  in  support  of  ongoing 
AFOSR  sponsored  research  in  the  Division  of  Applied  Mathematics  at  Brown  University.  The  primary 
purpose  of  the  equipment  is  to  enhance  computationally  intensive  research  in  high-order  numerical  methods 
and  high-performance  computing  currently  supported  under  AFOSR  Grant  AFOSR  F49620-99-1-0077,  High 
Order  Methods  for  the  Numerical  Simulation  of  High  Speed  Flows  (P.I.  -  Gottlieb,  Co-PI’s  -  W.S.  Don,  J.S. 
Hesthaven,  C.W.  Shu;  annual  funding  level,  $180K)  and  by  AFOSR/DARPA  (on  subcontract  to  HyperComp 
Inc.).  Advancing  the  Frontiers  of  Broad  Band  CEM  for  Modeling  Full-Scale  Treated  Targets  (PI  -  D.  Gottlieb; 
Co-PI  -  J.S.  Hesthaven;  Annual  funding  level,  $200K).  These  efforts  focuse  on  the  analysis  and  application 
of  novel  high-order  schemes,  designed  specifically  with  contemporary  parallel  high-performance  computer 
architectures  in  mind,  for  the  solution  of  wave  dominated  transient  problems  in  three-dimensional  complex 
geometries. 

This  work  entails  both  fundamental  and  applied  research  in  high-order  approximation  methods  (e.g., 
essentially  non-oscillatory  (ENO/WENO)  schemes,  spectral  methods,  spectral  element  methods,  and  dis¬ 
continuous  Galerkin  methods)  for  compressible  flow  simulations,  combustion,  electromagnetic  scattering  and 
penetration,  and  active  flow  control  for  flameholders  as  well  as  fundamental  and  applied  work  in  algorithms 
for  high-performance  parallel  computers. 

Typical  applications  include  three-dimensional  modeling  of  jet-engine  flameholders  and  shock  enhanced 
supersonic  combustion  and  mixing.  Problems  in  electromagnetics  include  broad-band  scattering  by  hetero¬ 
geneous  bodies  and  modeling  of  optical  devices. 

The  establishment  of  the  proposed  facility  paves  the  way  for  the  ability  to  transfer,  manipulate,  and 
visualize  very  large  temporal  datasets  resulting  from  the  simulation  of  two-  and  three-dimensional  transient 
phenomena  in  general  complex  geometries  as  those  currently  considered  as  part  of  the  AFOSR  sponsored 
research.  \\  it h  the  resources  available  prior  to  the  DURIP  award,  such  large  scale  manipulations  were  not 
possible,  hence  essentially  prohibiting  the  transfer  of  existing  computational  models  to  DOD  HPC  platforms 
due  to  an  inability  to  effectively  handle  the  very  large  datasets. 
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The  facility,  made  possible  by  this  grant,  was  designed  to  eliminate  this  bottleneck  and  facilitate  a  vastly 
,  extended  use  of  DOD  HPC  resources  to  solved  very  large  scale  unsteady  problems  using  high-order  methods 
as  part  of  AFOSR  supported  research  and  educational  efforts.  As  we  shall  describe  shortly,  this  has  already 
been  demonstrated  and  is  certain  to  be  invaluable  for  future  efforts. 

Equipment  Purchased 

A  list  of  purchased  equipment  and  approximate  expense  is  given  below. 


Qty. 

Item. 

total 

2 

Compaq  ES40  (4  750  MHz  Alpha  processors,  8GB  RAM) 

146,000 

2 

Myrinet  Interface  and  Cable  for  ES40 

2,690 

4 

SUN  Blade  1000  (2  750  MHz  Sparc  III  processors,  2GB  RAM) 

80,000 

4 

Fiber  Channel  Interface  for  SUN  Blade 

4,980 

5 

SUN  Ultra  10  (1  450  MHz  Sparc  II  processors,  1GB  RAM) 

12,915 

5 

Dell  PC  with  19?’  screen  (1  1.4  GHz  Pentium  IV  processors,  1GB  RAM) 

19,500 

1 

Xyratex  RAID  Array  w/  powersupply  and  FiberChannel  Interface 

13,235 

12 

Xyratex  73.4  GB 

13,560 

2 

Fiber  Channel  Cable 

170 

1 

Brocade  Fiber  Channel  Switch 

15,887 

12 

Brocade  Fiber  Channel  Cobber  Gigabit  Interconnect 

1,478 

4 

Brocade  Fiber  Channel  Fiber  Gigabit  Interconnect 

1,254 

1 

Brocade  Fiber  Channel  Switch  Rack 

235 

2 

SANergy  Interface  to  Compaq  ES40 

6,200 

4 

SANergy  Interface  to  SUN  Blade  1000 

12,400 

2 

SANergy  Professional  Installation 

3,000 

1 

Exabyte  X200  Archival  Tape  Library 

38,615 

100 

DLT  225m  Tapes 

7,925 

1 

Tivoli/IBM  Archival  Software 

10,000 

2 

Panasonic  DVD-RAM  for  PC 

1,090 

2 

PCI  Ultra  SCSI  Cards 

176 

1 

HP8500DN  Large  format  color  printer 

7,500 

2 

HP4100SHE  Laser  Printers 

2,300 

Total 

401,110 

inal  proposal  called  for  an  entirely  SUN  based  system.  However,  SUN  was  unable  to  offer  a  competitive  server 
solution  within  a  reasonable  timeframe.  The  server  plays  a  central  role  in  the  system  for  data-management 
and  processing,  price/performance  of  these  was  considered  the  main  priority,  hence  the  choice  of  the  Compaq 
SE40  solution.  As  the  main  role  of  these  machines  are  for  dataprocessing  and  manipulation  rather  than 
graphics  the  less  favorable  graphics  preformance  of  the  Compaq  solution  poses  no  problem  as  the  SUN  and 
highend  PC's  all  have  excellent  graphics  performance. 

The  originally  approved  award  was  on  $369,000  with  an  additional  $30,000  from  costsharing  by  Brown 
University,  totalling  $399,000.  The  remaining  $2,110  was  absorbed  by  Brown  University. 
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Overview  of  Facility 


The  purpose  of  the  facility  is  to  support  ongoing  research  and  educational  efforts  centered  around  the  devel¬ 
opment  and  application  of  novel  high-order  schemes,  exemplified  by  spectral  methods,  spectral  element  meth¬ 
ods  on  polymorphic  domains,  discontinuous  Galerkin  method  and  essentially  non-oscillatory  (ENO/WENO) 
schemes.  Applications  include  the  accurate  simulation  of  transient  phenomena  in  fluid  and  gas  flows,  com¬ 
bustion  problems,  electromagnetics  and  acoustics.  The  efforts  also  have  very  significant  components  of 
fundamental  and  applied  algorithmic  work  for  contemporary  high-performance  parallel  computers. 

The  problems  being  considered  in  these  efforts  are  characterized  by  a  significant  separation  of  scales  in 
time  and  space,  eigther  due  to  the  dynamics  of  the  problems  as  in  supersonic  gas-dynamics  and  combustion 
or  as  a  consequence  of  the  geometry  of  the  problems  as  exemplified  by  broad-band  electromagnetic  or  cLCOUStic 
scattering  by  large  heterogeneous  bodies.  In  most  cases  the  use  of  high-order  methods  is  a  necessity  to  obtain 
any  reliable  computational  results  for  such  problems. 

All  of  these  problems  are,  however,  also  characterized  by  a  very  significant  temporal  component  in  com¬ 
bination  with  a  truly  three-dimensional  character,  which  necessitates  the  use  of  remote  DOD  HPC  sites  due 
to  the  computational  complexity  of  a  problem  of  realistic  complexity 

The  facility  enables  a  much  needed  expansion  in  the  use  of  remote  DOD  HPC  resources  for  the  simulation 
of  two-  and  three-dimensional  transient  phenomena  in  general  complex  geometries  by  significantly  enhancing 
the  ability  to  transfer  as  well  as  pre/post-process,  visualize  and  animate  very  large  temporal  datasets. 

The  central  components  of  the  facility  are  the  two  Compaq  ES40  servers,  each  with  4  750  MHz  Alpha 
processors  and  8GB  of  RAM.  These  servers  provide  the  main  source  of  computing  power  and  ensure  the 
ability  to  handle  multiple  instances  of  large  datasets  for  processing,  feature  extraction  etc. 

Closely  connected  to  these  two  servers  are  4  SUN  Ultra  III  based  highend  workstations,  each  with  2  750 
MHz  Sparc  III  processors,  2GB  of  RAM  and  highend  graphics  capabilities.  These  machines  are  used  by 
senior  researchers  (faculty  and  postdocs)  with  extraordinary  needs  in  terms  of  computing  resources. 

To  support  the  ability  to  handle  large  datasets,  a  840GB  RAID  accelerated  discarray  has  been  purchased. 
This,  the  2  Compaq  servers  and  t he  4  SUN  Ultra  III  workstations  are  all  linked  together  in  a  dedicated  very 
broadband  storage  network  (SAN)  based  on  Fiber  Channel  links.  This  ensures  a  very  high  data  transfer  rate 
between  servers,  discs  and  workstations  for  efficient  data  manipulation  and  visualization.  The  workload  is 
split  in  such  a  way  that  the  workstations  perform  the  graphics  rendering  while  compute  intensive  pre-  and 
postprocessing  is  done  bv  the  servers.  The  SAN  is  also  connected  to  the  Brown  IBM  SP  to  serve  as  an  testing 
platform  as  well  as  to  the  vBNS  to  ensure  fast  and  reliable  transferral  of  data  to  and  from  DOD  HPC  sites. 

Initial  trials  with  the  graphics  package  Ensigth  Gold  suggested  that  this  would  serve  as  a  common  platform 
for  visualization.  However,  given  the  different  tasks  to  be  performed  we  have  chosen  to  use  Tecplot  as  the 
main  visualization  tool.  With  its  recent  support  of  OpenGL  this  is  found  to  give  very  good  performance,  a 
high  deg  ret*  of  user  friendliness  as  well  as  the  flexibility  needed  on  a  platform  serving  educational  as  well  as 
research  goals. 

A  second  ring  of  computing  harchvard  consists  of  5  highend  Linux  based  PC’s,  equipped  with  1.4  GHz 
Pentium  1\  processors.  1GB  RAM,  and  highend  graphics  cards,  as  well  as  5  SUN  Ultra  10,  Sparc  II  based 
workstations,  each  with  1GB  RAM.  These  10  machines  serve  the  students,  visitors,  and  faculty  involved  in 
the  sponsored  research.  They  are  all  connected  to  the  facility  by  100Mb  Ethernet.  Furthermore,  as  the  need 
arises  for  particular  individuals,  access  to  faster  and  more  closely  connected  hardware  as  described  above  will 
be  granted. 

I  be  large  disc  array  is  configured  with  a  large  temporary  space  which  it  is  the  users  respondsibility  to 
backup.  Regular  backup  is  only  offered  on  200GB  of  the  disc.  To  enable  this,  increased  capasity  of  the 
archival  tape  libraries  has  been  implemented  with  a  dedicated  archival  system  with  a  native  capasity  of  up 
to  11  IB.  Furthermore,  2  DVD-RAM'  systems  has  been  purchased  to  support  used  guided  data  transferral 
and  storage  as  well  as  for  storage  of  animations  etc. 

for  offline  data  analysis,  a  large  format  high  quality  color  laser  printer  has  been  purchased  as  well  as  two 
shared  laser  printers. 

1  ho  cent ral  parts  of  the  facility,  i.c..  the  two  Compaq  servers,  the  4  Sparc  III  workstations,  the  SAN 
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Figure  1:  Two-dimensional  computation  of  recessed  cavity  flameholder  flow  at  a  Reynolds  number  of  3.9*  107  (based 
on  the  free  stream  flow  and  a  unit  length)  and  a  Mach  number  of  1.91.  Shown  is  a  snapshot  of  the  water  fraction. 
The  model  is  the  full  compressible  Navier-Stokes  equations,  coupled  with  a  single  reversible  reaction  to  model  the 
combustion  of  hydrogen  in  an  oxygen/nitrogen  atmosphere. 


network  and  the  large  discarray,  are  logically  seperated  from  the  common  network  in  the  Division  and 
access  is  granted  only  to  students,  visitors,  and  faculty  directly  involved  in  AFOSR  sponsored  research  and 
educational  efforts. 

The  acquisitions  establishe  a  state-of-the-art  post-processing  and  visualization  facility  for  the  senior  inves¬ 
tigators  and  students  involved  in  the  current  AFOSR  sponsored  efforts.  Moreover,  it  establishes  a  powerful 
computational  platform  for  research-related  education  in  areas  of  research  relevant  to  the  missions  of  DOD 
and  will,  duo  to  the  emphasis  in  the  design  of  the  facility  on  flexibility,  expandability,  and  connectivity  to 
existing  resources,  have  an  extended  useful  lifetime. 

Impact  on  Applications 

In  the  following  we  shall  very  briefly  review  some  applications  already  taking  advantage  of  the  new  facility 
which,  in  some  cases,  represent  an  enabling  technology.  These  examples  are  not  meant  to  be  exhaustive  but 
rather  provide  illustrations  of  current  AFOSR  sponsored  efforts  benefitting  from  the  facility. 

The  applications  are  all  characterised  bv  inherent  time-dependence,  moderate  to  very  high  geometric 
complexity  and/or  the  apparence  of  strong  shock  and  near  singular  solutions. 

While  we  have  developed  and  tested  various  computational  schemes  over  the  last  years,  the  lack  of  powerful 
local  resources  for  the  very  demanding  transfer,  post-processing,  and  visualization  of  the  large  amounts  of 
data  resulting  from  these  computations  has  essentially  made  it  impossible  to  attempt  the  computation  of 
realistic  scenarios  using  DOD  HPC  resources. 

The  increased  local  bandwidth,  the  significant  local  storage,  and  the  enhanced  local  computing  power 
provided  by  the  dedicated  facility  has  been  central  to  avoid  that  the  data  analysis  be  the  limiting  factor 
in  the  modeling  of  these  types  of  problems.  The  reliable  high-speed  connections  to  DOD  HPC  through 
vBNS  permits  the  transferral  of  large  datasets  to  and  from  the  sites  and  the  local  proposed  facility  facilitates 
the  very  significant  task  of  manipulating,  visualizing  and  post-processing  the  multiple  instance  datasets  for 
realistic  three-dimensional  simulation. 

Problems  in  Supersonic  Mixing  and  Combustion 


Ongoing  work  on  high-order  modeling  of  supersonic  mixing  and  combustion  has  benefitted  significantly 
from  the  facility.  The  main  difficulty  associated  with  the  accurate  modeling  of  the  supersonic  mixing  and 
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Figure  2:  Shown  is  the  result  of  the  computation  where  Re  =  10000,  M  =  0.3,  using  6th  order  elements  and  a 
viscous  gridding.  The  left  illustrates  the  vorticity,  clearly  showing  the  von  Karman  street,  while  the  right,  illustrates 
the  identification  of  the  structures  computed  using  off-line  particle  tracking. 


combustion  is  the  extended  range  of  scales  that  co-exist  in  the  simulations  which  also  are  highly  temporal  in 
nature.  This  imposes  the  need  for  very  high  resolution  in  the  simulations,  typically  exceeding  10242  spectral 
modes  in  two-dimensions  and  at  least  51 23  modes  for  the  fully  three-dimensional  problem.  The  temporal 
nature  of  the  phenomena  requires  a  multiple  frame  analysis  to  recover  information  about  diffusion  rates,  the 
effects  of  chemical  reactions  etc. 

As  an  example  of  a  current  effort  we  consider  the  accurate  and  detailed  modeling  of  recessed  cavity 
fiameholders,  typically  operating  in  the  supersonic  flow-regime.  Recessed  cavity  flameholders  provide  a  high 
temperature,  low  speed  recirculation  zone  that  can  effectively  contain  the  production  of  radicals  during 
chemical  reactions  and  cavity  flameholders  are,  therefore,  being  considered  as  an  essential  component  for 
scram-jet  propulsion  systems  with  the  advantages  being  their  simplicity  and  their  low  total  pressure  losses 
for  steady  state  flows. 

To  model  the  flameholders  under  realistic  flow-conditions,  e.g.,  at  Mach  numbers  ranging  from  almost 
0  to  well  above  2.5  and  a  Reynolds  number  exceeding  105,  very  high  resolution  and  long  time  integration 
studies  is  required.  It  is  exactly  for  this  kind  of  problems  that  the  facility  has  been  designed,  providing  a 
front-end  and  post-processing  facility  to  large  scale  parallel  computations. 

In  Fig.  1  we  illustrate  the  results  of  a  recent  computation,  run  in  the  supersonic  regime  with  a  simple 
model  for  the  burning  of  hydrogen  in  a  oxygen /nitrogen  atmosphere.  While  these  results  are  obtained  using 
a  spectral  multidomain  code,  a  parallel  effort  involves  the  solution  of  the  same  problem  using  high-order 
WKNO  methods. 

We  are  using  experimental  data  from  WPAFB  to  verify  and  improve  the  computational  effort.  These 
experiments  dearly  illustrate  the  highly  temporal  and  three-dimensional  nature  of  the  very  complex  phe¬ 
nomena  and  it  is  only  through  this  kind  of  mutual  interaction  between  computational  modeling  and  detailed 
experimental  data  that  the  details  of  the  flow  scenarios  can  be  understood. 

Other  related  efforts,  benefiting  directly  from  the  purchased  equipment,  include  high-order  modeling 
of  Rayloigh-Taylor  and  Ritchmycr-Meshkov  instabilities  and  the  associated  mixing.  These  efforts  are  again 
guidr.l  by  solutions  obtained  with  different  schemes  as  well  as  by  comparison  with  experimental  results. 

Problems  in  Flameholder  Dynamics  and  Active  Control 

Another  ongoing  effort,  directly  and  immediately  benefitting  from  the  facility,  involves  the  modeling  of 
bluff  body  flameholders  with  the  aim  of  formulating  active  control  schemes  to  enhance  mixing.  The  typical 
scenario  is  characterized  by  a  bluff  body  sitting  within  a  narrow  channel.  The  bluff  body  gives  rice  to  the 
shedding  of  vortices,  hence  causing  the  flow  to  be  periodic  in  nature  —  at  least  for  low  Reynolds  numbers. 

The  computational  model  involves  a  high-order  unstructured  grid  method,  suitable  for  modeling  fully  com¬ 
pressible  visi  ons  g.Ls-dynarnics.  The  unstructured  grid  enables  t lie  accurate  representation  of  the  geometry, 
which  can  be  rather  complex  with  inlet-pipes  etc  for  control  of  the  flow,  while  the  high-order  accuracy  en¬ 
sures  an  accurate  and  reliable  solution.  The  implementation  is  fully  parallel  and  executes  well  on  distributed 
coinput  ing  plat  forms. 
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The  computations  serve  as  a  virtual  experiment  through  which  one  can  gain  understanding  of  the  flow 
pattern  and  develop  a  simple,  yet  efficient  active  control  loop  to  enhance  the  mixing  by  breaking  up  the 
structures  formed  rear  of  the  flameholder. 

In  Fig.  2  we  show  the  result  of  the  computation  with  Re  =  10000,  M  =  0.3,  using  6th  order  elements 
and  a  viscous  gridding.  One  clearly  observes  the  von  Karman  vortex  street  rear  of  the  bluff  body.  Figure  2 
also  shows  how  the  use  of  results  from  dynamical  systems  theory  allows  one  to  clearly  identify  the  structures. 
This  is  done  in  a  post-processing  phase,  utilizing  a  large  number  of  time-slices  to  track  particles  both  forward 
and  backward  in  time  to  identify  the  manifolds  associated  with  the  flow.  The  central  idea  is  that  one  who 
controls  the  structures,  controls  the  mixing.  What  is  also  shown  is  the  structures,  computed  using  massless 
tracers,  and  a  few  particle  paths  for  finite  mass  particles,  modeling  finite  size  fuel  drops  in  the  flow. 

This  effort  is  ongoing  and  will  continue  to  benefit  significantly  from  the  facility.  The  use  of  large  datasets 
in  which  to  track  particles  and  many  very  long  time  computations  to  understand  the  flowpattern  would 
not  have  been  possible  without  the  equipment.  With  this,  we  hope  to  develop  efficient  active  flow  control 
techniques  and  demonstrate  their  application  to  two-  and  three-dimensional  flows. 

Problems  in  Computational  Electromagnetics  and  Acoustics 

The  development  of  new  technologies  such  as  ultra  short  pulse  radars  and  integrated  optics  introduces 
an  increased  need  for  the  accurate  and  efficient  modeling  of  electromagnetic  problems  under  circumstances 
involving  very  broad  band  signals  and  often  electrically  large  objects  or  collections  thereof.  The  need  for 
the  full  vectorial  solution  of  the  electromagnetic  problem  subjected  to  a  general  time-dependent  excitation 
suggests  the  use  of  time-domain  methods  in  which  plane  wave  and  short  pulse  excitation  in  handled  equally 
by  directly  solving  the  full  vectorial  Maxwell  equations. 


Figure  3:  No  so  to  tail  component  of  the  reflected  magnetic  field  from  an  incident  plane  wave. 

To  address  such  problems  we  have  developed,  over  the  past  few  years,  a  high-order /spectral  time-domain 
solution  methods  for  Maxwells  equation  with  the  discretization  of  the  computational  domain  being  fully 
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unstructured  and  body  conforming.  One  of  the  novel  aspects  of  this  effort  is  the  representation  of  the 
approximate  solutions  using  truly  multi-variate  Lagrange  interpolation  polynomials  much  in  the  spirit  of 
classical  spectral  collocation  methods.  Maxwell's  equations  are  solved  in  a  discontinuous  element  formulation, 
ensure  a  highly  parallel  scheme  well  suited  for  large  scale  distributed  computing. 

As  an  illustration  of  the  ability  of  the  proposed  framework  to  address  problems  of  realistic  complexity,  we 
show  in  Fig.  3  an  example  of  electromagnetic  plane  wave  scattering  from  a  combat  fighter  plane.  The  finite 
element  grid  contains  123.000  elements  for  this  case. 

The  scheme  has  been  implemented  on  the  local  IBM  SP  as  well  as  on  the  NAVO  IBM  SP  for  large  scale 
computations.  Recently,  full  three-dimensional  computational  modeling  of  3.4  GHz  scattering  from  a  full 
aircraft  configuration  has  been  completed  using  this  code  on  the  NAVO  IBM  SP  and  up  to  512  processors. 
The  computations  are  done  at  6th  order  accuracy,  yielding  an  excess  of  iOa  degrees  of  freedom.  Each  snap 
shot  of  the  fields  exceeds  1GB  of  data. 

These  computations  and  data  manipulations  would  not  have  been  possible  without  the  facility  which  will 
be  instrumental  in  the  continuation  of  this  effort. 

Student  Training 


This  is  a  one-year  equipment  grant  which  primarily  addresses  infrastructure  improvements  for  scien¬ 
tific  computing  in  the  Division  of  Applied  Mathematics  at  Brown  and  as  such  does  not  support  individual 
researchers.  However,  the  group  involved  in  these  activities  currently  consists  of  3  faculty  members,  approx¬ 
imately  5  longterm  visitors  and  postdoctoral  researchers  as  well  as  an  average  of  10  graduate  students.  For 
most  of  these  individuals,  the  facility  will  continue  to  enhance  the  ability  to  conduct  research  central  to  DOD 
missions. 

Furthermore,  with  its  state-of-the-art  post-processing  and  visualization  capability,  the  facility  serves  as 
an  ideal  platform  for  research-related  education  in  a  number  of  HPC  related  areas  as  well  as  in  computational 
science,  numerical  analysis  etc.  Moreover,  the  vastly  improved  network  infrastructure  and  connectivity  to 
DOD  HPC  sites  ensures  reliable  access  to  DOD  facilities,  hence  allowing  the  students  to  exploit  and  take 
advantage  of  these  facilities  as  a  natural  ingredient  of  their  research  efforts. 


